{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Callbacks.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wc8qBUQaLF4",
        "colab_type": "text"
      },
      "source": [
        "# Implementing different Callbacks on IMDB Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ruKnS88taObG",
        "colab_type": "text"
      },
      "source": [
        "Here are some examples of ways you can use callbacks:\n",
        "\n",
        " Model checkpointing—Saving the current weights of the model at different points\n",
        "during training.\n",
        "\n",
        " Early stopping—Interrupting training when the validation loss is no longer\n",
        "improving (and of course, saving the best model obtained during training).\n",
        "\n",
        " Dynamically adjusting the value of certain parameters during training—Such as the\n",
        "learning rate of the optimizer.\n",
        "\n",
        " Logging training and validation metrics during training, or visualizing the representations learned by the model as they’re updated—The Keras progress bar that you’re\n",
        "familiar with is a callback!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAtV3h-daSTl",
        "colab_type": "text"
      },
      "source": [
        "The keras.callbacks module includes a number of built-in callbacks (this is not an\n",
        "exhaustive list):\n",
        "\n",
        "keras.callbacks.ModelCheckpoint\n",
        "\n",
        "keras.callbacks.EarlyStopping\n",
        "\n",
        "keras.callbacks.LearningRateScheduler\n",
        "\n",
        "keras.callbacks.ReduceLROnPlateau\n",
        "\n",
        "keras.callbacks.CSVLogger"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MA7cL-yXaU1P",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "a130b15c-6fee-471a-decb-f532bbdbbcb9"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.datasets import imdb\n",
        "from tensorflow.keras.preprocessing import sequence\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "max_features = 2000\n",
        "max_len = 500\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
        "\n",
        "x_train = sequence.pad_sequences(x_train, maxlen=max_len)\n",
        "x_test = sequence.pad_sequences(x_test, maxlen=max_len)\n",
        "\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(layers.Embedding(max_features, 128,input_length=max_len,name='embed'))\n",
        "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
        "model.add(layers.MaxPooling1D(5))\n",
        "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
        "model.add(layers.GlobalMaxPooling1D())\n",
        "model.add(layers.Dense(1))\n",
        "#model.summary()\n",
        "\n",
        "model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1u3p4X8kaZte",
        "colab_type": "text"
      },
      "source": [
        "# Early Stopping and model Checkpoint\n",
        "We can pass more than one callbacks in a list"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KpXA8asMaW0D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "callbacks_list = [tf.keras.callbacks.EarlyStopping(   # stop if not improving\n",
        "                monitor='acc',patience=1,),           # monitor validation accuracy\n",
        "                tf.keras.callbacks.ModelCheckpoint(\n",
        "                filepath='my_model.h5',\n",
        "                monitor='val_loss',                   # only save best weights\n",
        "                save_best_only=True,)]                # when vall loss is improved"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iww5MSR4acMP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "06e1b9e5-2c53-4401-abaf-a84cd0f647da"
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=callbacks_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "157/157 [==============================] - 75s 480ms/step - loss: 0.6174 - acc: 0.6837 - val_loss: 0.4078 - val_acc: 0.8308\n",
            "Epoch 2/2\n",
            "157/157 [==============================] - 76s 484ms/step - loss: 0.4369 - acc: 0.8480 - val_loss: 0.4545 - val_acc: 0.8592\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_gjV5v4alR-",
        "colab_type": "text"
      },
      "source": [
        "# THE REDUCELRONPLATEAU CALLBACK\n",
        "Use this callback to reduce the learning rate when the validation loss has\n",
        "stopped improving. Reducing or increasing the learning rate in case of a loss plateau to get out of local minima during training.\n",
        "###########################\n",
        "\n",
        "Platue straight surface hote hai aur gradiant us se bhr nahe araha hota\n",
        "qk gradiant ko neche jana hota hai aur woh surface sedhe hote hai"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mmy0V1zkakw9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "callbacks_list = [tf.keras.callbacks.ReduceLROnPlateau(\n",
        "                monitor='val_loss',factor=0.1,         # lr ko .1 se multiply kerdo (kam kerdo)\n",
        "                patience=10,)]                         # 10 epochs tak improvement nahe hoto stop hojai"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yFcvyfTocFkE",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "5bc3aee0-0ca3-4cf4-a655-bb993a22f668"
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=callbacks_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "157/157 [==============================] - 75s 481ms/step - loss: 0.3939 - acc: 0.8766 - val_loss: 0.4354 - val_acc: 0.8606\n",
            "Epoch 2/2\n",
            "157/157 [==============================] - 76s 483ms/step - loss: 0.3435 - acc: 0.9022 - val_loss: 0.6631 - val_acc: 0.8434\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pituU2xIexEC",
        "colab_type": "text"
      },
      "source": [
        "# We can create our own model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D-Ii6OBdey7I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "import numpy as np\n",
        "class ActivationLogger(keras.callbacks.Callback):\n",
        "            \n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        print(\"<<<<<< Epoch end >>>>>>>\")\n",
        "        \n",
        "        \n",
        "    def on_epoch_begin(self, epoch, logs=None):\n",
        "        print(\"<<<<<< Epoch begin >>>>>>>\")\n",
        "        \n",
        "        \n",
        "    def on_train_begin(self, epoch, logs=None):\n",
        "        print(\"<<<<<<< Training begin >>>>\")\n",
        "\n",
        "        \n",
        "    def on_train_end(self, epoch, logs=None):\n",
        "        print(\"<<<<<<< Training ended >>>>\")\n",
        "\n",
        "callbacks_list = [ActivationLogger()]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O8vWfWijgSRn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190
        },
        "outputId": "61dd82ee-039f-4d69-fa02-198b6f23c7d9"
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=callbacks_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<<<<<<< Training begin >>>>\n",
            "<<<<<< Epoch begin >>>>>>>\n",
            "Epoch 1/2\n",
            "157/157 [==============================] - ETA: 0s - loss: 0.2128 - acc: 0.9542<<<<<< Epoch end >>>>>>>\n",
            "157/157 [==============================] - 76s 484ms/step - loss: 0.2128 - acc: 0.9542 - val_loss: 0.7353 - val_acc: 0.8620\n",
            "<<<<<< Epoch begin >>>>>>>\n",
            "Epoch 2/2\n",
            "157/157 [==============================] - ETA: 0s - loss: 0.1788 - acc: 0.9698<<<<<< Epoch end >>>>>>>\n",
            "157/157 [==============================] - 77s 489ms/step - loss: 0.1788 - acc: 0.9698 - val_loss: 0.9845 - val_acc: 0.8460\n",
            "<<<<<<< Training ended >>>>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "434JUyHAjBcT",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# LearningRateScheduler\n",
        "\n",
        "seem same as REDUCELRONPLATEAU to me\n",
        "\n",
        "https://stackoverflow.com/questions/39779710/setting-up-a-learningratescheduler-in-keras"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Q_HmuPClz19",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 155
        },
        "outputId": "668a47bf-8c46-4276-b58c-9a7d627a3c5b"
      },
      "source": [
        "def lr_scheduler(epoch, lr):\n",
        "    decay_rate = 0.1\n",
        "    decay_step = 90\n",
        "    if epoch % decay_step == 0 and epoch:\n",
        "        return lr * decay_rate\n",
        "    return lr\n",
        "\n",
        "callbacks_list = [keras.callbacks.LearningRateScheduler(\n",
        "                  lr_scheduler, verbose=1)]\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=callbacks_list)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "Epoch 00001: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
            "Epoch 1/2\n",
            "157/157 [==============================] - 76s 481ms/step - loss: 0.1473 - acc: 0.9782 - val_loss: 0.8659 - val_acc: 0.8628\n",
            "\n",
            "Epoch 00002: LearningRateScheduler reducing learning rate to 0.0010000000474974513.\n",
            "Epoch 2/2\n",
            "157/157 [==============================] - 77s 488ms/step - loss: 0.1310 - acc: 0.9837 - val_loss: 0.9763 - val_acc: 0.8614\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ncFzYlEKmqgt",
        "colab_type": "text"
      },
      "source": [
        "Or if you  are not using GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yLFVzOXZlz49",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121
        },
        "outputId": "2ae4f4b3-f2f7-4e5b-8aaa-a81e5c902aa1"
      },
      "source": [
        "from tensorflow.keras.callbacks import LearningRateScheduler\n",
        "\n",
        "sd=[]\n",
        "class LossHistory(keras.callbacks.Callback):\n",
        "    def on_train_begin(self, logs={}):\n",
        "        self.losses = [1,1]\n",
        "\n",
        "    def on_epoch_end(self, batch, logs={}):\n",
        "        self.losses.append(logs.get('loss'))\n",
        "        sd.append(step_decay(len(self.losses)))\n",
        "        print('lr:', step_decay(len(self.losses)))\n",
        "\n",
        "def step_decay(losses):\n",
        "    if float(2*np.sqrt(np.array(history.losses[-1])))<0.3:\n",
        "        lrate=0.01*1/(1+0.1*len(history.losses))\n",
        "        momentum=0.8\n",
        "        decay_rate=2e-6\n",
        "        return lrate\n",
        "    else:\n",
        "        lrate=0.1\n",
        "        return lrate\n",
        "\n",
        "history=LossHistory()\n",
        "lrate=LearningRateScheduler(step_decay)\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=[history,lrate],verbose=2)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "lr: 0.1\n",
            "157/157 - 75s - loss: 7.6010 - acc: 0.5016 - val_loss: 7.7192 - val_acc: 0.4938\n",
            "Epoch 2/2\n",
            "lr: 0.1\n",
            "157/157 - 75s - loss: 7.6010 - acc: 0.5016 - val_loss: 7.7192 - val_acc: 0.4938\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VfnJtnialUJm",
        "colab_type": "text"
      },
      "source": [
        "# CSVLogger\n",
        "Callback that streams epoch results to a CSV file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-BOQ08jVlUy7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "95d30a8e-79bb-4aa9-f5fc-f682035c7ee5"
      },
      "source": [
        "from tensorflow.keras.callbacks import CSVLogger\n",
        "\n",
        "csv_logger = CSVLogger('log.csv', append=True, separator=';')\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    epochs=2,\n",
        "                    batch_size=128,\n",
        "                    validation_split=0.2,\n",
        "                    callbacks=[csv_logger])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/2\n",
            "157/157 [==============================] - 76s 486ms/step - loss: 7.6010 - acc: 0.5016 - val_loss: 7.7192 - val_acc: 0.4938\n",
            "Epoch 2/2\n",
            "157/157 [==============================] - 76s 485ms/step - loss: 7.6010 - acc: 0.5016 - val_loss: 7.7192 - val_acc: 0.4938\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iNmWU8a3p8f_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}